WORLD INTELLECTUAL PROPERTY ORGANIZATION 
International Bureau 




PCT 

INTERNATIONAL APPLICATION PUBLISHED UNDER THE PATENT COOPERATION TREATY (PCT) 



(51) International Patent Classification 7 : 
H04N 7/18 



Al 



(11) International Publication Number: WO 00/64176 

(43) International Publication Date: 26 October 2000 (26.10.00) 



(21) International Application Number: PCT/US00/10012 

(22) International Filing Date: 14 April 2000 (14.04.00) 



(30) Priority Data: 
60/129,812 



16 April 1999 (16.04.99) 



US 



(71) Applicant: PRINCETON VIDEO IMAGE, INC. [US/US]; 15 
Princess Road, Lawrenceville, NJ 08648 (US). 

(72) Inventor: ROSSER, Roy; 15 Princess Road, Lawrenceville, NJ 
08648 (US). 

(74) Agents: SOKOHL, Robert, E. et al.; Sterne, Kessler, Goldstein 
& Fox P.L.L.C., Suite 600, 1100 New York Avenue, N.W., 
Washington, DC 20005-3934 (US). 



(81) Designated States: AE, AG, AL, AM, AT, AU, AZ, BA, BB, 
BG, BR, BY, CA, CH, CN, CR, CU, CZ, DE, DK, DM, 
DZ, EE, ES, FI, GB, GD, GE, GH, GM, HR, HU, ID, IL, 
IN, IS, JP, KE, KG, KP, KR, KZ, LC, LK, LR, LS, LT, LU, 
LV, MA, MD, MG, MK, MN, MW, MX, NO, NZ, PL, PT, 
RO, RU, SD, SE, SG, SI, SK, SL, TJ, TM, TR, TT, TZ, 
UA, UG, UZ, VN, YU, ZA, ZW, ARIPO patent (GH, GM, 
KE, LS, MW, SD, SL, SZ, TZ, UG, ZW), Eurasian patent 
(AM, AZ, BY, KG, KZ, MD, RU, TJ, TM), European patent 
(AT, BE, CH, CY, DE, DK, ES, FI, FR, GB, GR, IE, IT, 
LU, MC, NL, PT, SE), OAPI patent (BF, BJ, CF, CG, CI, 
CM, GA, GN, GW, ML, MR, NE, SN, TD, TG). 



Published 

With international search report. 



(54) Title: METHOD AND APPARATUS TO OVERLAY COMPARATIVE TIME DETERMINED POSITIONAL DATA IN A VIDEO 
DISPLAY 

(57) Abstract 

In competitive events where competitors compete with each other over the same course at different times, the broadcast of the event 
is enhanced by displaying an icon of a first competitor's position on the course during the live video of the second competitor's run on 
the course. The position of the first competitor is monitored and stored up to at least sixty times per second of elapsed time of the first 
competitor's run. During the live video of the second competitor's run, camera sensors monitor the pan, tilt, focus, and zoom of the camera, 
thereby locating the field of view of the camera. The first competitor's position for the corresponding elapsed time of the second competitor 
is retrieved from storage. An indicia representing the first competitor's position at the corresponding elapsed time of the second competitor 
is inserted into the live video using a live video insertion system. This continuous update occurs up to sixty times per second such that the 
live video gives the appearance of the first competitor and the second competitor traversing the course at the same time. 
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Method and Apparatus to Overlay Comparative Time 
Determined Positional Data in a Video Display 

Background of the Invention 

Field of the Invention 

This invention relates to displaying comparative time-related positional 
data during video broadcasts or other forms of display of events in which 
competitors compete over the same course at different times; and in particular to 
means of showing video viewers how the competitors are performing relative to 
each other even though they are not competing over the same region of the course 
at the same time. 

Related Art 

Certain sporting events, such as downhill and slalom skiing, cycling time 
trials and auto-race qualifying, are run against the clock. Competitors typically 
compete by being individually timed over the same course to see who can cover 
the course in the shortest elapsed time. In televising these types of events, 
broadcasters typically show the elapsed time of the current competitor as a 
graphic overlay on the television picture. One way in which they make the 
competition more meaningful to the audience is to display how far behind or 
ahead of the leader the current competitor is at certain key timing points, such as 
the halfway mark. These points are often referred to as "split times". A 
shortcoming of this approach is that the viewer only has a real sense of how the 
competitor is faring in the competition at those key positions when the time 
difference between the leader and the current competitor is available for display 
and there are a limited number of those positions, typically only one or two. 
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Summary of the Invention 

The current invention aims to show in a graphic and continuous way, the 
relation between the current competitor and some previous competitor as they 
perform over an event course or portion of a racecourse. In most applications, the 

5 previous competitor would be the current event leader, but in practice it could be 

any previous competitor on whom data has been recorded or calculated 
performing on the same course or a similar course that could be extrapolated to 
match the current event course, including but not limited to a current or former 
record holder, a former title holder, the current competitor in a previous run or 

10 year, or even a fictitious competitor performing according to some bio-physical 

model or set of computations. 

There are two basic steps to implementing the invention. First, a record 
of a first competitor, whether it be the current leader or any previous competitor, 
must be developed by monitoring the first competitor's position as a continuous 

15 function of elapsed time, and storing the information in an easily recalled way. 

This makes it possible to give a continuously updated display of how far ahead 
or behind the leader the current competitor is. The second step is to then use a 
Live Video Insertion System to display the information in a form that is graphic 
and makes compelling television. A Live Video Insertion System is described in 

20 detail in U.S. Patent 5,264,933 to Rosser et aL, for example, the disclosure of 

which is incorporated in its entirety by reference herein. An example of one such 
graphic and compelling display would be to have an object such as a very visible 
dot, or other suitable graphic object or icon, continuously show where the first 
competitor would have been in the live shot of the current competitor. The 

25 graphic overlay could also be an image of the current leader shown in some 

stylized way such as with inverted coloring or ghosting. 

The first competitor, or graphic icon representing that competitor, could 
be used as a broadcast enhancement, or an advertiser could sponsor it. 
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The present invention is applicable to any event where competitors 
perform over substantially the same conditions. In addition to use in skiing, auto- 
racing, and cycling, as described above, the present invention could be used in 
other events including, but not limited to, track-and-field, swimming, speed 
5 skating, or even golf. In golf, for example, a previous competitor' s shot could be 

shown simultaneously with the present competitor's shot, using enhanced 
graphics to distinguish the two. Even a past champions shots could be shown by 
examining old film or video footage of the performance. 

Brief Description of the Figures 

10 The foregoing and other features and advantages of the invention will be 

apparent from the following, more particular description of a preferred 
embodiment of the invention, as illustrated in the accompanying drawings, 
where: 

Fig. 1 is a flowchart showing an overview of a method for showing a 
15 ghost competitor icon during a current competitor's live video. 

Fig. 2 is a schematic drawing showing how a ghost competitor icon could 
be implemented during a skiing competition; 

Fig. 3a shows a schematic of a data storage structure for recording object 
or competitor information; 
20 Fig. 3b shows a schematic of a data storage structure for recording camera 

location data; and 

Fig. 3c shows a schematic of a data storage structure for recording camera 
data for use in replays and post production. 
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Detailed Description of the Preferred Embodiments 

A preferred embodiment of the present invention is now described with 
reference to the figures where like reference numbers indicate identical or 
functionally similar elements. While specific configurations and arrangements 

5 are discussed, it should be understood that this is done for illustrative purposes 

only. A person skilled in the relevant art will recognize that other configurations 
and arrangements can be used without departing from the spirit and scope of the 
invention. It will be apparent to a person skilled in the relevant art that this 
invention can also be employed in a variety of other devices and applications. 

10 In particular, the present invention will be described using a ski event as 

an example. As explained above, and as will be apparent to a person skilled in 
the relevant art, the methods and apparatuses described can be applied equally to 
any event in which competitors compete one at a time or in sequence over a 
predefined or similar course, including, but not limited to cycling, athletics, auto 

15 racing, skiing, speed skating, in-line skating, street luge, luge, golf or 

bobsledding. It can also be applied to events in which competitors compete over 
a predefined course, but in different lanes, such as track events, swimming, horse 
racing, etc. The present invention can also be utilized for events such as rowing, 
for example, the Oxford Cambridge boat race or the Henley regatta, in which 

20 although there are two competitors racing over the same course, often the event 

becomes one sided after a very short time. Such events could be made more 
interesting for television viewers if an icon could be inserted showing how the 
current leaders are doing compared with a previous year's competitors or winners. 
Even in multi-competitor events such as track or swimming races, it would be of 

25 interest to the viewers to see where a previous year's winner or the current world, 

Olympic or national record holder would be at all stages in the current event. 
Golf is another such example, where the shots of competitors could be overlaid 
onto the live video of the current competitor' s shots. Even the shots taken by past 
great golfers at the same hole could be shown against each other or against the 
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current contender. The present invention can also be used to visually and 
graphically compare one golfer f s swing with the appropriately scaled and timed 
swings of other golfers, if the competitor is fitted with an appropriate monitoring 
setup. Further, in motor racing events such as Formula 1, as the race progresses 

5 and the contestants become strung out around the track, the ghost concept could 

be used to show how any of the trailing competitors is doing in terms of making 
or losing ground compared to how the race leader performed over the same parts 
of the course the last time he went over them. 

The general overview of a preferred method for overlaying comparative 

10 time determined positional data on a video display will be explained in 

accordance with Fig. 1. Fig. 1 shows a flow chart of a preferred method of the 
present invention. In this preferred method, block 100 represents the step of 
obtaining, recording, and storing positional and time elapsed data of a first 
competitors run along a course. Details of how this positional and time date is 

15 obtained, recorded, and stored will be discussed in more detail below, however, 

the basic concept provides that for every small fraction of time elapsed during the 
first competitors run, the position of the first competitor along the course is 
recorded and stored. Block 110 represents the start of the second competitor's run 
along the same course as the first competitor. This start triggers a time elapsed 

20 clock for the second competitor. As the second competitor's run is being shown 

live, block 120 represents retrieving the stored positional data of the first 
competitor for the corresponding elapsed time of the second competitor. 
Therefore, if the second competitor is exactly ten (10) seconds into the run, the 
position of the first competitor at ten (10) seconds into his/her run is retrieved 

25 from storage. Block 130 represents the step of inserting a graphic image of the 

position of the first competitor into the live video shot of the second competitor. 
The position of the first competitor retrieved from storage is calibrated to the field 
of view of the camera or cameras recording the second competitor's run along the 
course. Using this data, the position of the first competitor within the screen can 

30 be located precisely. Known video insertion techniques are then used to insert a 
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graphic image of the first competitor into the live video of the second competitor. 

Simultaneously with the live video of the second competitor and the 
insertion of a graphic image of the first competitor in the live video of the second 
competitor, the positional and time elapsed data of the second competitor is 

5 obtained, recorded and stored, as represented by block 140, similar to block 100 

for the first competitor. This data is stored in case the second competitor 
becomes the race leader, or if for some other reason the broadcaster wants to 
insert the second competitor's positional data into the live video of a third or 
subsequent competitor. 

10 The implementation of the invention as generally described above will be 

explained in detail referring to Fig. 2, which shows a ski course 210 on which a 
competitor 212 is competing and the event is being televised. 

In a preferred embodiment of the invention, competitor 212 ! s position in 
three dimensions is tracked in real time using a radio frequency (RF) local area 

15 multiple object tracking system. Such RF local area multiple object tracking 

systems are well known as discussed in International Application No. 
PCT/US98/03866 (published as Publication No. WO 98/37932) to Waddell et al , 
and in U.S. Patent Nos. 5,438,518 and 5,513,854, all of which are incorporated 
in their entireties by reference herein. Typically such systems consist of a well- 

20 known lightweight RF transmitter and antenna 214 worn by competitor 212. 

Such transmitters and antenna 214, including batteries, usually weigh only a few 
ounces and use well known spread spectrum technology to reduce power 
requirements and for efficacy in noisy RF environments. The signal from 
competitor 212's transmitter 214 is picked up by a plurality of RF receivers 216 

25 stationed in fixed locations around the course or arena in which the event is 

taking place. The signals received by the receivers 216 from the transmitter 214 
are relayed back to a central processing station 218, which may be a general- 
purpose computer programmed for the task. Using the different times of transit 
of signal to the different receivers, and knowing the locations of the receivers, it 

30 is possible to calculate the location of the transmitter using well known 
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algorithms and methods as discussed in detail in the Waddell et al. application 
referenced above and the references mentioned therein, each of which is 
incorporated in its entirety by reference herein. Central processing station 218 
uses the signals received to compute the three-dimensional position of competitor 

5 212, or more precisely, the competitor's transmitter 214, typically accurate to a 

within few centimeters and taken up to sixty times per second. The data 
recording competitor 212's position is then stored in a suitable storage unit 220, 
which is typically a commercially available magnetic disk storage disk. 

A typical competitor data structure 340 of the information stored 

10 regarding competitor 212 in storage unit 220 is shown schematically in Fig. 3a. 

Although there may be other, possibly more compact storage structures that will 
work, structure 300 represents the type of information that should be stored about 
the moving object or competitor, preferably at up to sixty times a second in order 
to have data with every field of NTSC video. Obviously if data is stored at lower 

15 data rates, positional data for any given field of video could be estimated by 

interpolation between stored data elements. The first item field 302 is a general 
event identification name or number, which may also encode the type of event 
being covered, the structure to be expected and the units each item will be stored 
in. Field 304 is the date of the event. Field 306 is an object or competitor 

20 identification name or number which may also include the particular run in a 

multi-run contest. Field 308 records the real time of day, which may be recorded 
in the standard broadcasting time of hours, minutes, seconds, frames of video and 
which field or may simply be the time of day accurate to hundredths of a second. 
Field 310 stores the elapsed time, which would typically be the time accurate to 

25 video fields or hundredths of a second from some timing fiducial such as the start 

of that competitor's run. An alternate method would be to record the start time 
in field 310 and calculate elapsed time by measuring the difference between the 
start time and the data time to calculate the elapsed time for that data set. Fields 
312, 314 and 316 then store the instantaneous positional data about that 

30 competitor or object. The units for the positional data is predefined and encoded 
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either in its own storage or in one of the identification names associated with the 
data. The scale, orientation and zero point of the data are effectively defined by 
the camera location data storage structure 320. Other three dimensional 
coordinate systems, such as the well known polar or radial coordinates, could also 
5 be used to store the data to equal effect, as would be understood by one of 

ordinary skill in the relevant art. 

Once the positional and time elapsed data for a first competitor 212 has 
been recorded and stored, a second competitor 212 starts a run along the same 
course as the first competitor. In a preferred embodiment of the invention, a 
10 camera 222 or a plurality of cameras 222 are used to televise the event. Cameras 

222 are equipped with camera sensors 224 which monitor the pan, tilt, zoom and 
focus of each camera 222. Such sensors are well known in the art and are 
discussed, for example, in co-pending International Application No. 
PCT/US97/21607, which is incorporated in its entirety by reference herein. 
15 Typically, camera sensors 224 comprise commercially available optical encoders, 

as supplied for instance by the Thoma company of Germany, which measure the 
pan and tilt of the camera with respect to a fixed tripod base. Additionally, zoom 
and focus data can be obtained directly from the drive mechanism of broadcast 
lenses. With calibration, this zoom and focus data can be interpreted with the 
20 data from the pan and tilt sensors to give the precise field of view of the camera, 

as described in detail in the aforementioned co-pending International Application 
No. PCT/US97/21607 or in pending International Application No. 
PCT/US96/1 1 122 (International Publication No. WO 97/02699) to Honey et al y 
which is also incorporated in its entirety by reference herein. 
25 In the preferred embodiment of the invention information from the camera 

sensors 224 is fed back along with the video feed from the camera or cameras 222 
to a live video insertion system 226. Live video insertion systems are well known 
and are described in detail in U.S. Patent Nos. 5,264,933, 5,543,856 and 
5,627,915 to Rosser et al y each of which is incorporated in its entirety by 
30 reference herein. Live video insertion systems are also described in the 
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aforementioned co-pending International Application No. PCT/US97/21607. 
Live video insertion systems generally operate on digital video, so if necessary, 
a first step is to convert the video into a digitized format. The system precisely 
determines the position within a field of the video in which an indicia will be 
inserted. Pattern recognition, as described in more detail below, may be used to 
aid in locating the insertion position. Next, the system determines if the indicia 
will occlude the object of interest within the video and, if so, appropriate 
occlusion masks are generated. The insertion indicia (e.g., icon or logo) is then 
warped to the correct size. The insertion indicia is then merged with the video, 
using any necessary occlusion masks, and the video is converted back to its 
original format. 

In a preferred embodiment of the present invention, the information from 
camera sensors 224 is fed back into live video insertion system 226 such that the 
field of view of the live feed from camera 222 is known. The positional data 
(from competitor data structure 300) of the first competitor corresponding to the 
exact elapsed time of the second competitor is retrieved. The positional data 
from the first competitor and the data regarding the field of view of camera 224 
are correlated such that the position of the first competitor is known within the 
field of view of camera 224. Using known live video insertion systems, the 
position of the first competitor is inserted into the live feed for the second 
competitor. Therefore, for every frame of NTSC video, the second competitor 
is shown live and the position of the first competitor is inserted into the video 
using a ghost competitor icon 232, thereby showing the viewers the position of 
the current competitor with respect to the previous competitor, such as the race 
leader. 

The data from the camera sensors is stored in a camera location data 
storage structure 320 as shown schematically in Fig. 3b. Camera location data 
storage structure 320 stores setup information about each camera 222 being used 
to televise the event on whose video stream ghost competitor icon 232 is going 
to inserted. Field 322 is a general event identification and field 324 is the date of 
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the event. These fields are equivalent to fields 302 and 304 of the competitor 
data structure 300 and are used to match up camera and competitor data files. 
Field 326 is used to store an identification which will link cameras 222 to a 
particular competitor or run in case camera locations are changed or the camera 

5 being used is movable. Field 328 is a camera identification name or number and 

may also have the camera and lens type and identification numbers coded in for 
accessing lens and camera calibration and characterization files that are used by 
the live video insertion system 226. Fields 330, 332 and 334 store the camera 
location in the same frame of reference, including coordinate system, scale and 

10 units, that the object data in the matching object data file 300 is stored. Fields 

336, 338, 340 and 342 store camera readings when the camera is set up to a 
standard locating shot, which may be of a standard reference object located in or 
alongside the event being televised. In the preferred embodiment, the fields 336, 
338, 340 and 342 contain the pan, zoom, tilt and focus data of the camera in units 

15 that are normally used by the live video insertion system software. The final four 

fields of this data structure 344, 346, 348 and 350 contain data about the standard 
reference object. Field 344 contains the name and type of the locator, for 
example, a standard six-foot black and white vertical pole framed to fill screen 
vertically. Fields 346, 348, and 350 contain the x, y and z positions of the 

20 reference object. The purpose of recording the pan, zoom, tilt and focus of the 

camera in some pre-set shot as well as the x, y and z coordinates of the object 
which is being viewed in the preset shot is to establish a scale and orientation to 
allow translation from the coordinate system of the local area object tracking 
system and the image coordinate system used by the live video insertion system 

25 226 being used to position the ghost competitor icon 232 correctly in the video 

being processed. 

For fixed cameras, there is only one set of camera location data 320. 
However, if handheld cameras are used and equipped with sensors that give 
instantaneous effective pan and tilt from a given orientation, zoom, focus and 
30 instantaneous x, y and z data, then the same data structure can be used to record 
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camera location and orientation at every video field with the addition of one more 
field recording actual time of day. 

Once the live video insertion system 226 has used the camera sensor data, 
either directly from camera 222 and camera sensor 224, or in a replay, from the 

5 camera data storage unit 225 along with the recorded video from video recording 

unit 223 to insert the appropriate icon stored in icon storage unit 227 into the 
video stream, the video stream is sent to a production switching unit 229, where 
it is mixed with the video and audio feeds from the other production cameras, 
recorders, microphones and graphics units, to produce the program. The program 

10 is then fed over distribution medium 231 to the viewers' video screen 230. Video 

storage unit 223 maybe, for example, a well known broadcast video tape recorder 
or broadcast quality video random access disk. Camera data storage unit 225 may 
be a well known digital hard disk or it may be stored as an integral part of storing 
the video, for example, by storing the data in the vertical blanking interval of the 

15 stored video. Distribution medium 231 may be any of the well known means of 

distributing video signals, including but not limited to radio frequency (RF) 
broadcast via transmitter towers or satellites, cable distribution via RF co-axial 
cable or optical fiber or over any suitable packet switching medium such a3, but 
not limited to, streaming video over high bandwidth internet connections. 

20 Utilizing the present invention, the positional data of a previous 

competitor or run captured from the local area object tracking system by central 
processing station 218 is stored on data storage unit 220 to enable live video 
insertion system 226 to seamlessly insert a ghost competitor icon 232 into the live 
video stream so that a viewer looking at screen 230 sees graphically where that 

25 competitor would have been relative to the current competitor had they both been 

competing at the same time. 

In order to provide a high quality integration of the previous competitors 
position into the live video of the current competitor, positional data from the 
previous run must be stored accurately and then synchronized with the current 

30 competitor's run. A preferred embodiment for synchronization is through a 
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master timing clock 215, which may be the standard black burst synchronization 
used in synching up broadcast video. This master timing signal or clock pulse is 
fed to both central processing station 218 used in recording the competitor's 
positions and live video insertion system 226, as well as video recording unit 223. 

5 Master timing clock 215 is also used to synchronize and relay the competitor or 

run start time from a start time device 213. Start time device 213 may simply be 
a button pressed by an official or it could be the same timing start device used in 
the official timing of the event. For example, start timing device 213 may be a 
beam of light that is broken by the competitor as he/she starts, a gate that is 

10 pushed back as the competitor goes through it, or a pressure pad that senses the 

competitor leaving the starting position. Other timing devices could also be used, 
as will be apparent to one of ordinary skill in the art. The time the event or run 
starts is recorded in the competitor data structure 300 and the camera data 
structures 320 and 360 (described in more detail below) as well as the real time, 

15 all synchronized to master timing clock 215 so that ghost competitor icon 232 

insertions can be accurately overlaid on each other with respect to elapsed time 
from start. 

Having all the data for the both the previous competitor and the current 
competitor, the three dimensional position on the course of the previous 

20 competitor with respect to time elapsed from the start can be inserted into the live 

video of the current competitor using live video insertion system 226, as 
discussed in detail in U.S. Patent Nos. 5,264,933; 5,543,856; 5,627,915 and 
5,892,554 and co-pending U.S. Application No. 09/230,099; Filed January 19, 
1999, the disclosures of which arc incorporated in their entirety by reference 

25 herein. Ghost competitor icon 232 can be a video of the previous competitor or 

simply a representative icon, appropriately sized and oriented, with occlusion 
where applicable, and inserted using the standard algorithms and methods of live 
video insertion as discussed in detail in the aforementioned references. 

The present invention can also be used for replays or using recorded 

30 footage. Fig. 3c shows a third data structure, camera data structure 360, that is 
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necessary if the insertion of ghost competitor icon 232 is going to be done on 
replays or if recorded footage is going to be used, partly or wholly as, or to create 
ghost competitor icon 232, or if the insertion of ghost competitor icon 232 is 
going to be done in post-production on recordings of the event. Camera data 
5 structure 360 is a way of synchronizing video that is being recorded from a 

particular camera with the data recording the position of an object after the event 
is over. It is also useful in synchronizing video from a camera with the position 
data of an object in situations where there may be significant (multi-field) delays 
in either data or video stream due to the transmission path of the data, such as, but 
10 not limited to, the case when the camera feed may be being relayed via a tower 

or satellite because of structures or terrain problems. The camera data may be 
encoded with the video it is associated with in, for example, the vertical blanking 
interval of the video signal or by any of the well know methods for encoding data 
in video signals as mentioned in U.S. Patent 5,543,856 to Rosser et a/., for 
15 example. Data field 362 stores the event identification name or number. Data 

field 364 stores the date. Data field 366 stores the camera identification name or 
number to allow synchronization of the camera data with the data stored \r. fV »^ 
camera location data structure 320. Data field 368 stores the real time, so that the 
camera data can be synchronized with the competitor data in competitor data 
20 structure 300. Data fields 370, 372, 374 and 376 are the camera pan, tilt, zoom 

and focus data, respectively. 

In the preferred embodiment explained above, the competitor and camera 
locations in three-dimensional space are determined using sensors 214 attached 
to the competitor 212 and cameras 222 and receiving antenna 218 to record 
25 distance. An alternative preferred embodiment of recording competitor 212 , s 

position is to use sensors 224 fitted on cameras 222. The camera operators in this 
embodiment must keep a part of the competitor at a cross hair or within a tightly 
defined box on the screen. The pan, tilt, zoom and focus data from a single 
camera at a given location can be used to determine where on a video image an 
30 object is, as discussed in detail in co-pending U.S. Application No. 09/230,099; 
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filed January 19,1 999, referenced above, and can be used to give the competitor's 
position with good accuracy in events where the competitors take a well defined 
line or stay within well defined lines, as is the case in most competitive sports. 
In this alternative embodiment, the first or previous competitor's positional data 
5 will still be stored in competitor data structure 300. However, this information 

is obtained from camera sensors 224 instead of the local area object tracking 
system. This embodiment eliminates the need for any transmitter 214 on the 
competitor, which may be important in events such as swimming and track and 
field, where only a few additional ounces may affect the competitor's 
10 performance. This embodiment also avoids costly set up of an RF local area 

object tracking system, which require receiver 216 and costly pre-event 
calibration. When using cameras 222 with sensors 224 to record the position of 
the competitors, the accuracy of the positional data can be improved by using 
multiple cameras 222. Using multiple cameras 222 also provides three- 
15 dimensional positioned data for the competitor. 

When using cameras 222 to capture the competitor's location data, it is 
important for accuracy purposes that the camera operators keep competitor 212 
within the cross-hairs or defined box of camera 222. In order to improve the 
accuracy of the location data captured by camera sensors 224, known pattern 
20 recognition techniques can be used. Pattern recognition techniques are disclosed, 

for example, in U.S. Patent 5,808,695 and International Application No. PCT 
US/97/21608, each of which is incorporated in its entirety by reference herein. 
Using this feature, a well defined portion of the competitor, for example a portion 
of a car in an auto race, will be identified for pattern recognition. If the cross- 
25 hairs of camera 222 trails or lags this defined portion, the system will utilize 

pattern recognition to record the correct location of the competitor even though 
the cross-hairs are not aligned exactly with the competitor. This provides for 
more accurate data capture of the competitor's location. 

Such known pattern recognition algorithms can also be used to aid in the 
30 seamless insertion of the ghost competitor icon 232 into the live video to make 
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it look as if the added indicia is a part of the original scene. In this embodiment 
of the invention, the camera sensor data is used as the search part of the live video 
insertion system, providing an initial starting point or location for the insertion 
of indicia. After the search phase, the indicia is maintained in the correct size 

5 shape and perspective to make it appear to be part of the original video scene by 

pattern recognition algorithms. Such algorithms are well known and are 
discussed in previously cited U.S. Patent 5,808,695 and International Application 
No. PCT US/97/21608. 

The present invention can also be used in multilane events such as 

10 swimming or track events. In these events, the lane the competitor on whom data 

is being collected would have to kept, and the distance from camera to the lanes 
would have to be calibrated. In events where the, competitors run in staggered 
lanes, such as the 200 and 400 meter sprint events in track and field, the lane 
number would also be important regardless of how the positional data is 

15 collected. The lane and stagger would then need to be calculate as an off-set in 

positioning ghost competitor icon 232. 

In another embodiment of the invention, the ins^on of ^o^t ^rpoptito: 
icon 232 could start as an image of a person or an object and morph or change 
into an icon. For instance, in a specific sporting event, an image of the current 

20 world record holder could appear as an insert at the start of the event. The image 

of the record holder could then be transformed into an icon which is then used to 
show the position of the world record holder during the event that is subsequently 
shown. As will be apparent to one of ordinary skill in the art, this morphing 
could take place at any time during the event. Ghost competitor icon 232 could 

25 also be used for advertising by using a logo, or it could morph into a logo at 

certain stages of the event. 

Ghost competitor icon 232 could also be inserted downstream, similar to 
the indicia being inserted at a downstream location as discussed in detail in U.S. 
Patent 5,543,856 and co-pending U.S. Patent Application No. 09/331,332; filed 

30 June 17, 1999, which is incorporated in its entirety by reference herein. In 
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particular, the data required for downstream insertion could be made available as 
a data bank to an end user, accessible or downloadable over the internet, so that 
the viewer could choose what or whom to overlay when watching the event 
broadcast over the internet on streaming video or on a digital receiver with access 

5 to the internet or networked to a computer that has access to the data base. 

If ghost competitor icon 232 is inserted in streaming video for distribution 
over the internet or other similar interactive media, the ghost insertion could be 
a hot button, as discussed in detail in co-pending U.S. Patent Application No. 
09/482,440; filed January 13, 2000, which is incoxporated in its entirety by 

10 reference herein. In particular,, the hot button ghost competitor could be a link to 

a home page that has details about, for instance, the competitor, or the record 
setting event that it is representing or the commercial sponsor. The hot button 
ghost competitor, when activated, could change the current insertion, or bring up 
a new web page with information or video or change the current streaming video. 

1 5 Alternatively, insertion of ghost competitor icon 232 could be done at the 

viewer's set top receiving device so that the viewer has the choice of having it 
displayed or not. 

If ghost competitor icon 232 is inserted in video for distribution in some 
multimedia recording format, for example, well known DVD format, ghost 

20 competitor icon 232 could be a hot link to another part of the recording medium, 

which could bring up video or information about the competitor, the record 
setting event or some associated commercial sponsor, for example. 

While the invention has been particularly shown and described with 
reference to preferred embodiments thereof, it will be understood by those skilled 

25 in the art that various changes in form and details may be made therein without 

departing from the spirit and scope of the invention. Thus the breadth and scope 
of the present invention should not be limited by any of the above-described 
exemplary embodiments, but should be defined only in accordance with the 
following claims and their equivalents. 
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What Is Claimed Is: 



1. A method of enhancing a broadcast of a sporting event, 
comprising the steps of: 

a) monitoring and storing a position of a first competitor in a 
5 sporting event with respect to time using camera sensor data, wherein said stored 

position at each unit in time is stored in a separate data structure; 

b) receiving a video image from a camera, wherein a second 
competitor is within a field of view of said camera; 

c) acquiring camera sensor data from said camera while said second 
10 competitor is within a field of view of the camera, said camera sensor data 

representing the position and orientation of said field of view of said camera; 

d) retrieving said stored position of said first competitor for the 
corresponding time that has elapsed since the second competitor has started said 
sporting event; and 

15 e) inserting an indicia into said video image based on said stored 

position data of said first competitor and said field of view of said camera. 



2. The method of claim 1, wherein data is stored in a data structure 
sixty times per second, and wherein said data structure includes a time of day 
field, an elapsed time field, and positional data fields. 
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3. The method of claim 1 , further comprising creating a second data 
structure that stores a location of said camera prior to the start of said event, 
wherein said second data structure includes a camera location field, pan, zoom, 
tilt and focus fields, and a plurality of fields relating to a standard reference 

5 object. 

4. The method of claim 3, further comprising storing a plurality of 
data structures identical to said second data structure, wherein said plurality of 
data structures represent the multiple positions of a handheld camera. 

5. The method of claim 1 , further comprising the step of monitoring 
10 and storing aposition of said second in the sporting event with respect to the time 

using said camera sensor data, wherein said stored position at each unit in time 
is stored in a separate data structure. 

6. The method of claim 1, wherein the step of inserting an indicia 
into said video image comprises utilizing a live video insertion system. 

15 7. The method of claim 1, wherein said indicia is an icon. 

8. The method of claim 7, wherein said icon includes sponsorship 
information. 

9. The method of claim 1 , wherein said indicia is a ghost image of 
said first competitor. 



20 10. The method of claim 1, wherein said video image is transmitted 

through a video stream to the internet. 
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11. The method of claim 10, wherein said indicia comprises a hot 

button. 

12. The method of claim 1 1 , wherein when said hot button is activated 
by a viewer, information about the first competitor is displayed. 

5 13. The method of claim 1 1 , wherein when said hot button is activated 

by a viewer, information about a sponsor is displayed. 

14. The method of claim 1 , wherein said indicia morphs from an icon 
to an image. 

15. The method of claim 1, wherein said indicia morphs from an 
10 image to an icon. 

16. An apparatus for enhancing a broadcast of a sporting event, 
comprising: 

a camera including camera sensors, said camera tracking a first competitor 
and said camera sensors monitoring the pan, tilt, zoom, and focus of said camera 
15 to monitor and store a position of said first competitor with respect to time, 

wherein said stored position at each unit in time is stored in a separate data 
structure; 

means for receiving a video image from said camera when a second 
competitor is within a field of view of said camera, said camera sensors providing 
20 camera sensor data representative representing the position and orientation of a 

field of view of said camera; 

means for retrieving the stored position of said first competitor for the 
corresponding elapsed time of said second competitor; and 

means for inserting an indicia into said video image based on the position 
25 data of said first competitor and said field of view of said camera. 
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17. The apparatus of claim 16, wherein data is stored in said first data 
structure sixty times per second, and wherein said first data structure includes a 
time of day field, an elapsed time field, and positional data fields. 

18. The apparatus of claim 1 6, further comprising means for creating 
5 a second data structure that stores a location of said camera prior to the start of 

said event, wherein said second data structure includes a camera location field, 
pan, zoom, tilt and focus fields, and a plurality of fields relating to a standard 
reference object. 

19. The apparatus of claim 18, further comprising means for storing 
10 a plurality of data structures identical to said second data structure, wherein said 

plurality of data structures represent the multiple positions of a handheld camera. 

20. The apparatus of claim 16, wherein said camera and said camera 
sensors monitor and store a position of said second competitor with respect to 
time, wherein said stored position at each unit in time is stored in a separate data 

15 structure. 

21. The apparatus of claim 16, wherein said means for inserting an 
indicia into said video image comprises utilizing a live video insertion system. 

22. The apparatus of claim 16, wherein said indicia is an icon. 

23. The apparatus of claim 22, wherein said icon includes sponsorship 
20 information. 



24. The apparatus of claim 16, wherein said indicia is a ghost image 
of said first competitor. 
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25. The apparatus of claim 16, wherein said video image is transmitted 
through a video stream to the internet. 

26. The apparatus of claim 25, wherein said indicia comprises a hot 

button. 

27. The apparatus of claim 26, wherein when said hot button is 
activated by a viewer, information about the first competitor is displayed. 

28. The apparatus of claim 26, wherein when said hot button is 
activated by a viewer, information about a sponsor is displayed. 

29. The method of claim 1 6, wherein said indicia morphs from an icon 
to an image. 

30. The method of claim 16, wherein said indicia morphs from an 
image to an icon. 
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